{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分析A/B测试结果\n",
    "\n",
    "这个项目可以帮你确认你已经掌握了统计课程中涵盖的所有内容。 希望这个项目尽可能地涵盖所有内容。 祝你好运！\n",
    "\n",
    "## 目录\n",
    "- [简介](#intro)\n",
    "- [I - 概率](#probability)\n",
    "- [II - A/B 测试](#ab_test)\n",
    "- [III - 回归](#regression)\n",
    "\n",
    "\n",
    "<a id='intro'></a>\n",
    "### 简介\n",
    "\n",
    "通常情况下，A/B 测试由数据分析师和数据科学家来完成。如果你在一些实践工作中遇到过这方面的问题，那学习起来就会更加游刃有余。\n",
    "\n",
    "对于这个项目，你将要了解的是电子商务网站运行的 A/B 测试的结果。你的目标是通过这个 notebook 来帮助公司弄清楚他们是否应该使用新的页面，保留旧的页面，或者应该将测试时间延长，之后再做出决定。\n",
    "\n",
    "**使用该 notebook 的时候，请同步学习课堂内容，并回答与每个问题相关的对应测试题目。** 每个课堂概念的标签对应每个题目。这样可以确保你在完成项目的过程中的方法正确，并且你最终提交的内容会更加符合标准，不必担心出现错误。最后检查的时候，请确保你的提交内容符合 [审阅标准](https://review.udacity.com/#!/projects/37e27304-ad47-4eb0-a1ab-8c12f60e43d0/rubric) 中的所有标准。\n",
    "\n",
    "<a id='probability'></a>\n",
    "#### I - 概率\n",
    "\n",
    "让我们先导入库，然后开始你的任务吧。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "#We are setting the seed to assure you get the same answers on quizzes as we set up\n",
    "random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`1.` 现在，导入 `ab_data.csv` 数据，并将其存储在 `df` 中。  **使用你的 dataframe 来回答课堂测试 1 中的问题。**\n",
    "\n",
    "a. 导入数据集，并在这里查看前几行："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp    group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739  control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739  control     old_page          0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('ab-data.csv')\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. 使用下面的单元格来查找数据集中的行数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "294478"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. 数据集中独立用户的数量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.user_id.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. 用户转化的比例。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11965919355605512"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(df.converted == 1).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(treatment    147276\n",
       " control      147202\n",
       " Name: group, dtype: int64, new_page    147239\n",
       " old_page    147239\n",
       " Name: landing_page, dtype: int64)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.group.value_counts(), df.landing_page.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e.  `new_page` 与 `treatment` 不一致的次数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3893"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df[((df.group == 'treatment')&(df.landing_page == 'old_page'))])+len(df[((df.group == 'control')&(df.landing_page == 'new_page'))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "f. 是否有任何行存在缺失值？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id         0\n",
       "timestamp       0\n",
       "group           0\n",
       "landing_page    0\n",
       "converted       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` 对于 **treatment** 不与 **new_page** 一致的行或 **control** 不与 **old_page** 一致的行，我们不能确定该行是否真正接收到了新的或旧的页面。我们应该如何处理这些行？在课堂中的 **测试 2** 中，给出你的答案。  \n",
    "\n",
    "a. 现在，使用测试题的答案创建一个符合测试规格要求的新数据集。将新 dataframe 存储在 **df2** 中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290585"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = df[((df.group == 'treatment') == (df.landing_page == 'new_page'))].copy()\n",
    "len(df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Double Check all of the correct rows were removed - this should be 0\n",
    "df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.user_id.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`3.` 使用 **df2** 与下面的单元格来回答课堂中的 **测试3** 。\n",
    "\n",
    "a.  **df2** 中有多少唯一的 **user_id**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.user_id.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b.  **df2** 中有一个重复的 **user_id** 。它是什么？ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>2893</td>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2.user_id.duplicated()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. 这个重复的  **user_id** 的行信息是什么？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([2893], dtype='int64')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2.user_id.duplicated()]['user_id'].index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. 删除 **一个** 含有重复的 **user_id** 的行， 但需要确保你的 dataframe 为 **df2**。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.drop(index = 2893, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`4.` 在下面的单元格中，使用 **df2** 来回答与课堂中的 **测试 4** 相关的测试题目。\n",
    "\n",
    "a. 不管它们收到什么页面，单个用户的转化率是多少？\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.converted.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. 假定一个用户处于 `control` 组中，他的转化率是多少？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1203863045004612"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.query('group == \"control\"').converted.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. 假定一个用户处于 `treatment` 组中，他的转化率是多少？\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11880806551510564"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.query('group ==\"treatment\" ').converted.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. 一个用户收到新页面的概率是多少？\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5000619442226688"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(df2.landing_page == 'new_page').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. 使用这个问题的前两部分的结果，给出你的建议：你是否认为有证据表明一个页面可以带来更多的转化？在下面写出你的答案。\n",
    "\n",
    "**这里我们可以看出，因为control组（对照组）与treatment组（实验组）的转化率几乎相同，所以无法判断一个新页面带来更多转化。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ab_test'></a>\n",
    "### II - A/B 测试\n",
    "\n",
    "请注意，由于与每个事件相关的时间戳，你可以在进行每次观察时连续运行假设检验。  \n",
    "\n",
    "然而，问题的难点在于，一个页面被认为比另一页页面的效果好得多的时候你就要停止检验吗？还是需要在一定时间内持续发生？你需要将检验运行多长时间来决定哪个页面比另一个页面更好？\n",
    "\n",
    "一般情况下，这些问题是A / B测试中最难的部分。如果你对下面提到的一些知识点比较生疏，请先回顾课程中的“描述统计学”部分的内容。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`1.` 现在，你要考虑的是，你需要根据提供的所有数据做出决定。如果你想假定旧的页面效果更好，除非新的页面在类型I错误率为5％的情况下才能证明效果更好，那么，你的零假设和备择假设是什么？ 你可以根据单词或旧页面与新页面的转化率 **$p_{old}$** 与 **$p_{new}$** 来陈述你的假设。\n",
    "\n",
    "**零假设是旧的页面更好，备选假设是新的页面更好，所以，应当是Pnew > Pold，也就是说Pnew - Pold > 0。假设u = Pnew - Pold，那么零假设是u<=0，备选假设是u>0。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` 假定在零假设中，不管是新页面还是旧页面， $p_{new}$ and $p_{old}$ 都具有等于 **转化** 成功率的“真”成功率，也就是说，  $p_{new}$ 与 $p_{old}$ 是相等的。此外，假设它们都等于**ab_data.csv** 中的 **转化** 率，新旧页面都是如此。  <br><br>\n",
    "\n",
    "每个页面的样本大小要与 **ab_data.csv** 中的页面大小相同。  <br><br>\n",
    "\n",
    "执行两次页面之间 **转化** 差异的抽样分布，计算零假设中10000次迭代计算的估计值。  <br><br>\n",
    "\n",
    "使用下面的单元格提供这个模拟的必要内容。如果现在还没有完整的意义，不要担心，你将通过下面的问题来解决这个问题。你可以通过做课堂中的 **测试 5** 来确认你掌握了这部分内容。<br><br>\n",
    "\n",
    "a. 在零假设中，$p_{new}$ 的 **convert rate（转化率）** 是多少？\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pnew = df2.converted.mean()\n",
    "pnew"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. 在零假设中， $p_{old}$  的 **convert rate（转化率）** 是多少？ <br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pold = df2.converted.mean()\n",
    "pold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c.  $n_{new}$ 是多少？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145310"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nnew = len(df2.query('landing_page ==\"new_page\" '))\n",
    "nnew"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d.  $n_{old}$?是多少？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145274"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nold = len(df2.query('landing_page ==\"old_page\" '))\n",
    "nold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. 在零假设中，使用 $p_{new}$ 转化率模拟 $n_{new}$ 交易，并将这些 $n_{new}$ 1's 与 0's 存储在 **new_page_converted** 中。(提示：可以使用  [numpy.random.choice](https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.choice.html)。)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_page_converted = np.random.choice(a =[0,1], size = nnew, p = [1-pnew, pnew])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. 在零假设中，使用 $p_{old}$ 转化率模拟 $n_{old}$ 交易，并将这些  $n_{old}$ 1's 与 0's 存储在 **old_page_converted** 中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "old_page_converted = np.random.choice(a = [0,1], size = nold, p = [1-pold, pold])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "g. 在 (e) 与 (f)中找到 $p_{new}$ - $p_{old}$ 模拟值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0014957558309978969"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_page_converted.mean() - old_page_converted.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "h. 使用**a. 到 g. ** 中的计算方法来模拟 10,000个 $p_{new}$ - $p_{old}$ 值，并将这 10,000 个值存储在 **p_diffs** 中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_diffs = []\n",
    "\n",
    "for _ in range(10000):\n",
    "    new_page_converted_mean = np.random.choice(a = [0,1], size = nnew, replace = True, p = [1-pnew, pnew]).mean()\n",
    "    old_page_converted_mean = np.random.choice(a = [0,1], size = nold, replace = True, p = [1-pold, pold]).mean()\n",
    "    p_diffs.append(new_page_converted_mean - old_page_converted_mean)        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "i. 绘制一个 **p_diffs** 直方图。这个直方图看起来像你所期望的吗？通过回答课堂上的匹配问题，确保你完全理解这里计算出的内容。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAASLElEQVR4nO3df6xf9X3f8eerJpBtSYsphnm2M7upJ9X8UZJZhCn7g5UODFQxlRbJSG2sFMmVBlKitaqc5g+6dEikXUsXLaVyi1WnS0tYkygWYaMuTVVVGj9MSgjGZb4BGm7sgTtTkioak7P3/vh+3H65/t57v773fu/3Jp/nQzr6nvM+n3PO52NLr3vuOed7bqoKSVIfvm/aHZAkrR5DX5I6YuhLUkcMfUnqiKEvSR25aNodWMjll19eW7dunXY3JOm7ylNPPfXXVbVh1Lo1Hfpbt27l6NGj0+6GJH1XSfJX863z8o4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqyKKhn+StSZ5I8pUkx5L8+1bfluTxJCeSfCbJxa1+SVueaeu3Du3rI63+fJIbJzUoSdJo45zpvwH8WFX9KHA1sCvJtcDHgXurajvwGnB7a3878FpV/TBwb2tHkh3AHuAqYBfwm0nWreRgJEkLWzT0a+Bv2+Jb2lTAjwF/2OqHgFvb/O62TFt/fZK0+gNV9UZVvQjMANesyCgkSWMZ6xu57Yz8KeCHgU8CXwP+pqrOtiazwKY2vwl4GaCqziZ5HfjBVn9saLfD2wwfax+wD+Ad73jHBQ5Hvdm6/4tTO/ZL99wytWNLSzXWjdyq+k5VXQ1sZnB2/iOjmrXPzLNuvvrcYx2oqp1VtXPDhpGvjpAkLdEFPb1TVX8D/ClwLXBpknO/KWwGTrb5WWALQFv/A8CZ4fqIbSRJq2Ccp3c2JLm0zf8D4MeB48CXgH/Tmu0FvtDmD7dl2vo/qcEf4j0M7GlP92wDtgNPrNRAJEmLG+ea/kbgULuu/33Ag1X1UJLngAeS/AfgL4D7W/v7gd9LMsPgDH8PQFUdS/Ig8BxwFrijqr6zssORJC1k0dCvqmeAd42ov8CIp2+q6v8A759nX3cDd194NyVJK8Fv5EpSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkfGerWypPNN67XOvtJZy+GZviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR1ZNPSTbEnypSTHkxxL8qFW/6Uk30jydJtuHtrmI0lmkjyf5Mah+q5Wm0myfzJDkiTNZ5w/onIW+Lmq+nKStwNPJTnS1t1bVf9xuHGSHcAe4CrgnwB/nOSftdWfBP41MAs8meRwVT23EgORJC1u0dCvqlPAqTb/rSTHgU0LbLIbeKCq3gBeTDIDXNPWzVTVCwBJHmhtDX1JWiUXdE0/yVbgXcDjrXRnkmeSHEyyvtU2AS8PbTbbavPV5x5jX5KjSY6ePn36QronSVrE2KGf5G3AZ4EPV9U3gfuAdwJXM/hN4NfONR2xeS1Qf3Oh6kBV7ayqnRs2bBi3e5KkMYz1h9GTvIVB4H+6qj4HUFWvDK3/beChtjgLbBnafDNwss3PV5ckrYJxnt4JcD9wvKp+fai+cajZTwLPtvnDwJ4klyTZBmwHngCeBLYn2ZbkYgY3ew+vzDAkSeMY50z/vcBPA19N8nSr/SJwW5KrGVyieQn4WYCqOpbkQQY3aM8Cd1TVdwCS3Ak8AqwDDlbVsRUciyRpEeM8vfPnjL4e//AC29wN3D2i/vBC20mSJstv5EpSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SerIoqGfZEuSLyU5nuRYkg+1+mVJjiQ50T7Xt3qSfCLJTJJnkrx7aF97W/sTSfZObliSpFHGOdM/C/xcVf0IcC1wR5IdwH7g0araDjzalgFuAra3aR9wHwx+SAB3Ae8BrgHuOveDQpK0OhYN/ao6VVVfbvPfAo4Dm4DdwKHW7BBwa5vfDXyqBh4DLk2yEbgROFJVZ6rqNeAIsGtFRyNJWtAFXdNPshV4F/A4cGVVnYLBDwbgitZsE/Dy0GazrTZffe4x9iU5muTo6dOnL6R7kqRFjB36Sd4GfBb4cFV9c6GmI2q1QP3NhaoDVbWzqnZu2LBh3O5JksYwVugneQuDwP90VX2ulV9pl21on6+2+iywZWjzzcDJBeqSpFUyztM7Ae4HjlfVrw+tOgycewJnL/CFofoH2lM81wKvt8s/jwA3JFnfbuDe0GqSpFVy0Rht3gv8NPDVJE+32i8C9wAPJrkd+Drw/rbuYeBmYAb4NvBBgKo6k+SXgSdbu49V1ZkVGYUkaSyLhn5V/Tmjr8cDXD+ifQF3zLOvg8DBC+mgJGnl+I1cSeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0JekjozzamVpUVv3f3HaXZA0Bs/0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjiwa+kkOJnk1ybNDtV9K8o0kT7fp5qF1H0kyk+T5JDcO1Xe12kyS/Ss/FEnSYsY50/9dYNeI+r1VdXWbHgZIsgPYA1zVtvnNJOuSrAM+CdwE7ABua20lSato0bdsVtWfJdk65v52Aw9U1RvAi0lmgGvaupmqegEgyQOt7XMX3GNJ0pIt55r+nUmeaZd/1rfaJuDloTazrTZf/TxJ9iU5muTo6dOnl9E9SdJcSw39+4B3AlcDp4Bfa/WMaFsL1M8vVh2oqp1VtXPDhg1L7J4kaZQl/RGVqnrl3HyS3wYeaouzwJahppuBk21+vrokaZUs6Uw/ycahxZ8Ezj3ZcxjYk+SSJNuA7cATwJPA9iTbklzM4Gbv4aV3W5K0FIue6Sf5A+A64PIks8BdwHVJrmZwieYl4GcBqupYkgcZ3KA9C9xRVd9p+7kTeARYBxysqmMrPhpJ0oLGeXrnthHl+xdofzdw94j6w8DDF9Q7SdKK8hu5ktQRQ1+SOmLoS1JHDH1J6oihL0kdWdKXsyRNz9b9X5zasV+655apHVsrwzN9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1ZNHQT3IwyatJnh2qXZbkSJIT7XN9qyfJJ5LMJHkmybuHttnb2p9Isncyw5EkLWScM/3fBXbNqe0HHq2q7cCjbRngJmB7m/YB98HghwRwF/Ae4BrgrnM/KCRJq2fR0K+qPwPOzCnvBg61+UPArUP1T9XAY8ClSTYCNwJHqupMVb0GHOH8HySSpAlb6jX9K6vqFED7vKLVNwEvD7WbbbX56udJsi/J0SRHT58+vcTuSZJGWekbuRlRqwXq5xerDlTVzqrauWHDhhXtnCT1bqmh/0q7bEP7fLXVZ4EtQ+02AycXqEuSVtFSQ/8wcO4JnL3AF4bqH2hP8VwLvN4u/zwC3JBkfbuBe0OrSZJW0UWLNUjyB8B1wOVJZhk8hXMP8GCS24GvA+9vzR8GbgZmgG8DHwSoqjNJfhl4srX7WFXNvTksSZqwRUO/qm6bZ9X1I9oWcMc8+zkIHLyg3kmSVpTfyJWkjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR25aNod0Mrauv+L0+6CpDXMM31J6oihL0kdWVboJ3kpyVeTPJ3kaKtdluRIkhPtc32rJ8knkswkeSbJu1diAJKk8a3Emf6/qqqrq2pnW94PPFpV24FH2zLATcD2Nu0D7luBY0uSLsAkLu/sBg61+UPArUP1T9XAY8ClSTZO4PiSpHksN/QL+KMkTyXZ12pXVtUpgPZ5RatvAl4e2na21d4kyb4kR5McPX369DK7J0kattxHNt9bVSeTXAEcSfKXC7TNiFqdV6g6ABwA2Llz53nrJUlLt6wz/ao62T5fBT4PXAO8cu6yTft8tTWfBbYMbb4ZOLmc40uSLsySQz/JP0ry9nPzwA3As8BhYG9rthf4Qps/DHygPcVzLfD6uctAkqTVsZzLO1cCn09ybj+/X1X/PcmTwINJbge+Dry/tX8YuBmYAb4NfHAZx5YkLcGSQ7+qXgB+dET9fwPXj6gXcMdSjydJWj6/kStJHTH0Jakjhr4kdcRXK0sa27Re3f3SPbdM5bjfizzTl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xD+iMgHT+kMTkrQYz/QlqSOGviR1xNCXpI4Y+pLUEUNfkjqy6k/vJNkF/CdgHfA7VXXPavdB0neXaT4R99I9t0zt2JOwqmf6SdYBnwRuAnYAtyXZsZp9kKSerfaZ/jXATFW9AJDkAWA38NwkDubz8pKWa1o5MqnfMFY79DcBLw8tzwLvGW6QZB+wry3+bZLnV6lv47gc+Otpd2JKeh47OP6exz+Vsefjy9r8n863YrVDPyNq9aaFqgPAgdXpzoVJcrSqdk67H9PQ89jB8fc8/u+1sa/20zuzwJah5c3AyVXugyR1a7VD/0lge5JtSS4G9gCHV7kPktStVb28U1Vnk9wJPMLgkc2DVXVsNfuwTGvystMq6Xns4Ph7Hv/31NhTVYu3kiR9T/AbuZLUEUNfkjrSfegnuSzJkSQn2uf6edrtbW1OJNk7VP/nSb6aZCbJJ5JkznY/n6SSXD7psSzFpMaf5FeT/GWSZ5J8PsmlqzWmxSTZleT51uf9I9ZfkuQzbf3jSbYOrftIqz+f5MZx97mWrPT4k2xJ8qUkx5McS/Kh1RvNhZvE/39bty7JXyR5aPKjWIaq6noCfgXY3+b3Ax8f0eYy4IX2ub7Nr2/rngD+BYPvIPw34Kah7bYwuGn9V8Dl0x7rao4fuAG4qM1/fNR+pzTedcDXgB8CLga+AuyY0+bfAr/V5vcAn2nzO1r7S4BtbT/rxtnnWpkmNP6NwLtbm7cD/7On8Q9t9++A3wcemvY4F5q6P9Nn8BqIQ23+EHDriDY3Akeq6kxVvQYcAXYl2Qh8f1X9jxr8r39qzvb3Ar/AnC+grTETGX9V/VFVnW3bP8bgOxlrwd+9CqSq/i9w7lUgw4b/Tf4QuL79BrMbeKCq3qiqF4GZtr9x9rlWrPj4q+pUVX0ZoKq+BRxn8O37tWgS//8k2QzcAvzOKoxhWQx9uLKqTgG0zytGtBn1+ohNbZodUSfJ+4BvVNVXJtHpFTSR8c/xMwx+C1gL5hvLyDbtB9frwA8usO04+1wrJjH+v9MuhbwLeHwF+7ySJjX+32Bwgvf/Vr7LK6uLP4ye5I+Bfzxi1UfH3cWIWs1XT/IP275vGHP/E7Xa459z7I8CZ4FPj3msSVu0zwu0ma8+6uRprf52N4nxDzZK3gZ8FvhwVX1zyT2crBUff5KfAF6tqqeSXLfM/k1cF6FfVT8+37okryTZWFWn2uWKV0c0mwWuG1reDPxpq2+eUz8JvJPBNb+vtPuam4EvJ7mmqv7XMoayJFMY/7l97wV+Ari+Xf5ZC8Z5Fci5NrNJLgJ+ADizyLbfLa8Xmcj4k7yFQeB/uqo+N5mur4hJjP99wPuS3Ay8Ffj+JP+lqn5qMkNYpmnfVJj2BPwqb76R+Ssj2lwGvMjgJub6Nn9ZW/ckcC1/fyPz5hHbv8TavZE7kfEDuxi8MnvDtMc4ZywXMbgRvY2/v5F31Zw2d/DmG3kPtvmrePONvBcY3BhcdJ9rZZrQ+MPgfs5vTHt80xj/nG2vY43fyJ16B6Y9MbhW9yhwon2eC7OdDP6y17l2P8Pgxs0M8MGh+k7gWQZ38v8z7VvOc46xlkN/IuNv7V4Gnm7Tb017rEN9vpnBEyZfAz7aah8D3tfm3wr81zaGJ4AfGtr2o22753nzk1rn7XOtTis9fuBfMrj88czQ//d5Jz9rZZrE///Q+jUf+r6GQZI64tM7ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR15P8DtdVeWRLjpkoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(p_diffs);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "j.  在**p_diffs**列表的数值中，有多大比例大于 **ab_data.csv** 中观察到的实际差值？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0015782389853555567"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_diffs = np.array(p_diffs)\n",
    "obs_diff = df2.query('landing_page == \"new_page\"')['converted'].mean() - df2.query('landing_page == \"old_page\"')['converted'].mean()\n",
    "obs_diff\n",
    "# 这里顺序不能错，前边是new-old,后边自然也就是new-old。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9051"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(p_diffs > obs_diff).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "k. 用文字解释一下你刚才在 **j.**中计算出来的结果。在科学研究中，这个值是什么？ 根据这个数值，新旧页面的转化率是否有区别呢？\n",
    "\n",
    "**这个值是P值，由于该P值大于0.05，这种情况下我们无法拒绝零假设。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "l. 我们也可以使用一个内置程序 （built-in）来实现类似的结果。尽管使用内置程序可能更易于编写代码，但上面的内容是对正确思考统计显著性至关重要的思想的一个预排。填写下面的内容来计算每个页面的转化次数，以及每个页面的访问人数。使用 `n_old` 与 `n_new` 分别引证与旧页面和新页面关联的行数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "convert_old = len(df2.query('landing_page ==\"old_page\" and converted ==\"1\" '))\n",
    "convert_new = len(df2.query('landing_page ==\"new_page\" and converted ==\"1\" '))\n",
    "n_old = len(df2.query('landing_page == \"old_page\"'))\n",
    "n_new = len(df2.query('landing_page == \"new_page\"'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "m. 现在使用 `stats.proportions_ztest` 来计算你的检验统计量与 p-值。[这里](http://knowledgetack.com/python/statsmodels/proportions_ztest/) 是使用内置程序的一个有用链接。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.3109241984234394, 0.9050583127590245)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sm.stats.proportions_ztest(count = [convert_old, convert_new], nobs = [n_old, n_new], alternative = 'smaller')\n",
    "# 使用该定义时，count表示的是需要表明的部分，而nobs表示的是总量，最后alternative表示的是方法，需要注意的是，这三个值之间有联系，需要一一队应，如果\n",
    "# 前边是new，old，那么后边也应该是new，old，而我们的备择假设是pnew>pold，所以自然也是“larger”。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "n. 根据上题算出的 z-score 和 p-value，我们认为新旧页面的转化率是否有区别？它们与 **j.** 与 **k.** 中的结果一致吗？\n",
    "\n",
    "**根据P值大小，由于远大于0.05，所以我们认为新旧页面的转化率几乎没有区别，与之前的**j.** 与 **k.** 中的结果一致。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='regression'></a>\n",
    "### III - 回归分析法之一\n",
    "\n",
    "`1.` 在最后一部分中，你会看到，你在之前的A / B测试中获得的结果也可以通过执行回归来获取。<br><br>\n",
    "\n",
    "a. 既然每行的值是转化或不转化，那么在这种情况下，我们应该执行哪种类型的回归？\n",
    "\n",
    "**我们应当执行逻辑回归。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. 目标是使用 **statsmodels** 来拟合你在 **a.** 中指定的回归模型，以查看用户收到的不同页面是否存在显著的转化差异。但是，首先，你需要为这个截距创建一个列（ 原文：column） ，并为每个用户收到的页面创建一个虚拟变量列。添加一个 **截距** 列，一个 **ab_page** 列，当用户接收 **treatment** 时为1， **control** 时为0。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  ab_page  \n",
       "0          1        0  \n",
       "1          1        0  \n",
       "2          1        1  \n",
       "3          1        1  \n",
       "4          1        0  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['intercept'] = 1\n",
    "df2[['control','ab_page']] = pd.get_dummies(df2.group)\n",
    "df2.drop(columns = ['control'], inplace = True)\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "c. 使用 **statsmodels** 导入你的回归模型。 实例化该模型，并使用你在 **b.** 中创建的2个列来拟合该模型，用来预测一个用户是否会发生转化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "sm_logit = sm.Logit(df2['converted'], df2[['intercept','ab_page']])\n",
    "results = sm_logit.fit();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. 请在下方提供你的模型摘要，并根据需要使用它来回答下面的问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                 <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290582</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>   <td>     1</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Fri, 10 Jan 2020</td> <th>  Pseudo R-squ.:     </th>  <td>8.077e-06</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>13:32:53</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th>   <td>0.1899</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9888</td> <td>    0.008</td> <td> -246.669</td> <td> 0.000</td> <td>   -2.005</td> <td>   -1.973</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0150</td> <td>    0.011</td> <td>   -1.311</td> <td> 0.190</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290582\n",
       "Method:                           MLE   Df Model:                            1\n",
       "Date:                Fri, 10 Jan 2020   Pseudo R-squ.:               8.077e-06\n",
       "Time:                        13:32:53   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "Covariance Type:            nonrobust   LLR p-value:                    0.1899\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9888      0.008   -246.669      0.000      -2.005      -1.973\n",
       "ab_page       -0.0150      0.011     -1.311      0.190      -0.037       0.007\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. 与 **ab_page** 关联的 p-值是多少？ 为什么它与你在 **II** 中发现的结果不同？<br><br>  **提示**: 与你的回归模型相关的零假设与备择假设分别是什么？它们如何与 **Part II** 中的零假设和备择假设做比较？\n",
    "\n",
    "**p-值是0.190，不同的原因是：首先这里的零假设和备择假设与之前的回归模型中的零假设和备择假设不同：总体斜率等于0的零假设以及参数不等于 0 的备择假设；另外就是双方的计算方法不一样：II中使用的是A/B测试，而小节中的逻辑回归使用了最大似然估计**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. 现在，你一定在考虑其他可能影响用户是否发生转化的因素。讨论为什么考虑将其他因素添加到回归模型中是一个不错的主意。在回归模型中添加附加项有什么弊端吗？\n",
    "\n",
    "**首先，可以确定一个或者多个相对变量对反应变量影响的能力，再者，可以识别出异常值；但是也存在以下弊端：反应变量和新添的附加选项之间可能不存在线性关系；附加选项中可能具有相关性误差；可能没有恒定常数方差；可能附加项内含有离群点，来破坏整个模型；可能会有更多的线性度；所有这些都是附加项可能存在的潜在弊端；**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "g. 现在，除了测试不同页面的转化率是否会发生变化之外，还要根据用户居住的国家或地区添加一个 effect 项。你需要导入 **countries.csv** 数据集，并将数据集合并在适当的行上。 [这里](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.join.html) 是链接表格的文档。 \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>country</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  ab_page country  CA  UK  US  \n",
       "0          1        0      US   0   0   1  \n",
       "1          1        0      US   0   0   1  \n",
       "2          1        1      US   0   0   1  \n",
       "3          1        1      US   0   0   1  \n",
       "4          1        0      US   0   0   1  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_country = pd.read_csv('countries.csv')\n",
    "df_new = df2.join(df_country.set_index('user_id'),on = 'user_id' )\n",
    "df_new = df_new.join(pd.get_dummies(df_new.country)) \n",
    "df_new.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366116\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                 <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290581</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>   <td>     2</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Fri, 10 Jan 2020</td> <th>  Pseudo R-squ.:     </th>  <td>1.521e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>13:32:57</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th>   <td>0.1984</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9967</td> <td>    0.007</td> <td> -292.314</td> <td> 0.000</td> <td>   -2.010</td> <td>   -1.983</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA</th>        <td>   -0.0408</td> <td>    0.027</td> <td>   -1.518</td> <td> 0.129</td> <td>   -0.093</td> <td>    0.012</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>    0.0099</td> <td>    0.013</td> <td>    0.746</td> <td> 0.456</td> <td>   -0.016</td> <td>    0.036</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290581\n",
       "Method:                           MLE   Df Model:                            2\n",
       "Date:                Fri, 10 Jan 2020   Pseudo R-squ.:               1.521e-05\n",
       "Time:                        13:32:57   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "Covariance Type:            nonrobust   LLR p-value:                    0.1984\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9967      0.007   -292.314      0.000      -2.010      -1.983\n",
       "CA            -0.0408      0.027     -1.518      0.129      -0.093       0.012\n",
       "UK             0.0099      0.013      0.746      0.456      -0.016       0.036\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_new['intercept'] = 1\n",
    "sm_logit_country = sm.Logit(df_new['converted'], df_new[['intercept','CA','UK']])\n",
    "result_country = sm_logit_country.fit()\n",
    "result_country.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个国家项对转化有影响吗？不要忘记为这些国家的列创建虚拟变量—— **提示: 你将需要为这三个虚拟变量增加两列。** 提供统计输出，并书面回答这个问题。\n",
    "\n",
    "**根据两者P值来看，国家对转化应该没有什么影响。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "h. 虽然你现在已经查看了国家与页面在转化率上的个体性因素，但现在我们要查看页面与国家/地区之间的相互作用，测试其是否会对转化产生重大影响。创建必要的附加列，并拟合一个新的模型。  \n",
    "\n",
    "提供你的摘要结果，以及根据结果得出的结论。\n",
    "\n",
    "**提示：页面与国家/地区的相互作用**\n",
    "```\n",
    "df3['new_CA'] = df3['new_page'] * df3['CA']\n",
    "df3['new_UK'] = df3['new_page'] * df3['UK']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>country</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "      <th>new_CA</th>\n",
       "      <th>new_UK</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  ab_page country  CA  UK  US  new_CA  new_UK  \n",
       "0          1        0      US   0   0   1       0       0  \n",
       "1          1        0      US   0   0   1       0       0  \n",
       "2          1        1      US   0   0   1       0       0  \n",
       "3          1        1      US   0   0   1       0       0  \n",
       "4          1        0      US   0   0   1       0       0  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_new['new_CA'] = df_new['ab_page'] * df_new['CA']\n",
    "df_new['new_UK'] = df_new['ab_page'] * df_new['UK']\n",
    "df_new.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366109\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                 <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290578</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>   <td>     5</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Fri, 10 Jan 2020</td> <th>  Pseudo R-squ.:     </th>  <td>3.482e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>13:32:59</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th>   <td>0.1920</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9865</td> <td>    0.010</td> <td> -206.344</td> <td> 0.000</td> <td>   -2.005</td> <td>   -1.968</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0206</td> <td>    0.014</td> <td>   -1.505</td> <td> 0.132</td> <td>   -0.047</td> <td>    0.006</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA</th>        <td>   -0.0175</td> <td>    0.038</td> <td>   -0.465</td> <td> 0.642</td> <td>   -0.091</td> <td>    0.056</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>   -0.0057</td> <td>    0.019</td> <td>   -0.306</td> <td> 0.760</td> <td>   -0.043</td> <td>    0.031</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>new_CA</th>    <td>   -0.0469</td> <td>    0.054</td> <td>   -0.872</td> <td> 0.383</td> <td>   -0.152</td> <td>    0.059</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>new_UK</th>    <td>    0.0314</td> <td>    0.027</td> <td>    1.181</td> <td> 0.238</td> <td>   -0.021</td> <td>    0.084</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290578\n",
       "Method:                           MLE   Df Model:                            5\n",
       "Date:                Fri, 10 Jan 2020   Pseudo R-squ.:               3.482e-05\n",
       "Time:                        13:32:59   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "Covariance Type:            nonrobust   LLR p-value:                    0.1920\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9865      0.010   -206.344      0.000      -2.005      -1.968\n",
       "ab_page       -0.0206      0.014     -1.505      0.132      -0.047       0.006\n",
       "CA            -0.0175      0.038     -0.465      0.642      -0.091       0.056\n",
       "UK            -0.0057      0.019     -0.306      0.760      -0.043       0.031\n",
       "new_CA        -0.0469      0.054     -0.872      0.383      -0.152       0.059\n",
       "new_UK         0.0314      0.027      1.181      0.238      -0.021       0.084\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_new['intercept'] = 1\n",
    "sm_logit_country = sm.Logit(df_new['converted'], df_new[['intercept','ab_page','CA','UK','new_CA','new_UK']])\n",
    "result_country = sm_logit_country.fit()\n",
    "result_country.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**根据以上P值得知，他们都与转化率没有什么关系。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='conclusions'></a>\n",
    "## 总结\n",
    "\n",
    "根据上面的测试结果，我们可以得到判断：不推荐这家电商使用新的页面。\n",
    "\n",
    "### Resources\n",
    "\n",
    "https://sciencing.com/advantages-disadvantages-multiple-regression-model-12070171.html\n",
    "\n",
    "https://blog.csdn.net/u010161630/article/details/52012756"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
